Scalable space-optimized and energy-efficient computing system

ABSTRACT

A scalable space-optimized and energy-efficient computing system is provided. The computing system comprises a plurality of modular compartments in at least one level of a frame configured in a hexadron configuration. The computing system also comprises an air inlet, an air mixing plenum, and at least one fan. In the computing system the plurality of modular compartments are affixed above the air inlet, the air mixing plenum is affixed above the plurality of modular compartments, and the at least one fan is affixed above the air mixing plenum. When at least one module is inserted into one of the plurality of modular compartments, the module couples to a backplane within the frame.

BACKGROUND

The present application relates generally to an improved data processingapparatus and method and more specifically to mechanisms for a scalablespace-optimized and energy-efficient computing system.

High-performance computing (HPC) uses supercomputers and computerclusters to solve advanced computation problems. The HPC term is mostcommonly associated with computing used for scientific research. Arelated term, high-performance technical computing (HPTC), generallyrefers to the engineering applications of cluster-based computing (suchas computational fluid dynamics and the building and testing of virtualprototypes). Recently, HPC has come to be applied to business uses ofcluster-based supercomputers, such as data intensive, commercialanalytics applications, and transaction processing.

However, many HPC systems are hindered by limits in the powerconsumption, space, cooling, and adaptability. That is HPC systems arecomposed out of thousands of components which occupy considerable space,require considerable cooling, use massive power, and are not readablydeployable.

SUMMARY

In one illustrative embodiment, a scalable space-optimized andenergy-efficient computing system is provided. In the illustrativeembodiment, the computing system comprises a plurality of modularcompartments in at least one level of a frame configured in a hexahedronconfiguration. In the illustrative embodiment, the computing system alsocomprises an air inlet, an air mixing plenum, and at least one fan. Inthe illustrative embodiment, the plurality of modular compartments areaffixed above the air inlet. In the illustrative embodiment, the airmixing plenum is affixed above the plurality of modular compartments. Inthe illustrative embodiment, the at least one fan is affixed above theair mixing plenum. In the illustrative embodiment, when at least onemodule is inserted into one of the plurality of modular compartments,the module couples to a backplane within the frame.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exampleembodiments of the present invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The invention, as well as a preferred mode of use and further objectivesand advantages thereof, will best be understood by reference to thefollowing detailed description of illustrative embodiments when read inconjunction with the accompanying drawings, wherein:

FIG. 1 depicts a pictorial representation of an example distributed dataprocessing system in which aspects of the illustrative embodiments maybe implemented;

FIG. 2 shows a block diagram of an example data processing system inwhich aspects of the illustrative embodiments may be implemented;

FIG. 3 depicts one exemplary configuration of a symmetric multiprocessor(SMP) system in the form of a processor node in accordance with anillustrative embodiment;

FIG. 4A depicts an example of one side of a processing module inaccordance with an illustrative embodiment;

FIG. 4B depicts an exemplary cubical processing module in accordancewith an illustrative embodiment;

FIG. 4C depicts an exemplary storage module in accordance with anillustrative embodiment;

FIG. 4D depicts an exemplary input/output (I/O) module in accordancewith an illustrative embodiment;

FIG. 4E depicts an exemplary filler module in accordance with anillustrative embodiment;

FIG. 5A depicts an exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment;

FIG. 5B depicts an example of a constructed processing module thatillustrates the majority of the air space within the middle of aprocessing module being filled in accordance with an illustrativeembodiment;

FIG. 6A depicts another exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment;

FIG. 6B illustrates removal of a heatsink and circuit board from aprocessing module side in accordance with an illustrative embodiment;

FIG. 6C depicts a core that may be inserted into an empty space in thecenter of a processing module in accordance with an illustrativeembodiment;

FIG. 6D illustrates another embodiment of an expanding core inaccordance with an illustrative embodiment;

FIG. 7A depicts an example of a scalable space-optimized andenergy-efficient computing system in accordance with an illustrativeembodiment;

FIG. 7B depicts an exemplary frame of a ubiquitous high-performancecomputing (UHPC) system in accordance with an illustrative embodiment;

FIG. 7C depicts an exemplary top down view of a UHPC system inaccordance with an illustrative embodiment;

FIG. 7D depicts an exemplary a view of the cooling fans of a UHPC systemin accordance with an illustrative embodiment;

FIG. 8A depicts an exemplary module that is liquid tight and liquidcooled in order to increase heat dissipation in accordance with anillustrative embodiment;

FIG. 8B depicts an exemplary cooling of multiple modules by a singleheat exchanger in accordance with an illustrative embodiment;

FIG. 8C depicts an exemplary structure where the center area of a UHPCsystem is populated by one or more modules;

FIG. 8D depicts how a three-dimensional very-large-scale integration(VLSI) global routing technique may be applied to generate a layout forthe tubes in accordance with an illustrative embodiment;

FIG. 8E depicts another exemplary structure where the heat exchangersalong the walls of a module are replaced by heat exchanger panels inaccordance with an illustrative embodiment; and

FIG. 8F depicts a heat exchanger panel in accordance with anillustrative embodiment.

DETAILED DESCRIPTION

The illustrative embodiments provide a ubiquitous high-performancecomputing (UHPC) system that packages the thousands of components of ahigh-performance computing (HPC) system into building-block modules thatmay be coupled together to form a space-optimized and energy-efficientproduct. The illustrative embodiments also provide for various heatsinkdesigns that enable an elegant assembly and in place maintenance for theheatsink and the module, while maintaining large effective heat exchangearea and high pressure for efficient cooling. The illustrativeembodiments also provide for an alternative to air cooling using aliquid cooling system with coolant/air heat exchanging enabled by skinheat exchangers mounted either on the interior or the exterior surfaceof the UHPC system.

Thus, the illustrative embodiments may be utilized in many differenttypes of data processing environments including a distributed dataprocessing environment, a single data processing device, or the like. Inorder to provide a context for the description of the specific elementsand functionality of the illustrative embodiments, FIGS. 1-3 areprovided hereafter as example environments in which aspects of theillustrative embodiments may be implemented. While the descriptionfollowing FIGS. 1-3 will focus primarily on a single data processingdevice implementation of a ubiquitous high-performance computing (UHPC)system, this is only an example and is not intended to state or implyany limitation with regard to the features of the present invention. Tothe contrary, the illustrative embodiments are intended to includedistributed data processing environments and embodiments in which aubiquitous high-performance computing (UHPC) system may easily beimplemented.

With reference now to the figures and in particular with reference toFIGS. 1-3, example diagrams of data processing environments are providedin which illustrative embodiments of the present invention may beimplemented. It should be appreciated that FIGS. 1-3 are only examplesand are not intended to assert or imply any limitation with regard tothe environments in which aspects or embodiments of the presentinvention may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

With reference now to the figures, FIG. 1 depicts a pictorialrepresentation of an example distributed data processing system in whichaspects of the illustrative embodiments may be implemented. Distributeddata processing system 100 may include a network of computers in whichaspects of the illustrative embodiments may be implemented. Thedistributed data processing system 100 contains at least one network102, which is the medium used to provide communication links betweenvarious devices and computers connected together within distributed dataprocessing system 100. The network 102 may include connections, such aswire, wireless communication links, or fiber optic cables.

In the depicted example, ubiquitous high-performance computing (UHPC)server 104 and server 106 are connected to network 102 along withstorage unit 108. In addition, clients 110, 112, and 114 are alsoconnected to network 102. These clients 110, 112, and 114 may be, forexample, personal computers, network computers, or the like. In thedepicted example. UHPC server 104 provides data, such as boot files,operating system images, and applications to the clients 110, 112, and114. Clients 110, 112, and 114 are clients to UHPC server 104 in thedepicted example. Distributed data processing system 100 may includeadditional servers, clients, and other devices not shown.

In the depicted example, distributed data processing system 100 is theInternet with network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, the distributed data processing system 100 may also beimplemented to include a number of different types of networks, such asfor example, an intranet, a local area network (LAN), a wide areanetwork (WAN), or the like. As stated above, FIG. 1 is intended as anexample, not as an architectural limitation for different embodiments ofthe present invention, and therefore, the particular elements shown inFIG. 1 should not be considered limiting with regard to the environmentsin which the illustrative embodiments of the present invention may beimplemented.

With reference now to FIG. 2, a block diagram of an example dataprocessing system is shown in which aspects of the illustrativeembodiments may be implemented. Data processing system 200 is an exampleof a computing system in which computer usable code or instructionsimplementing the processes for illustrative embodiments of the presentinvention may be located.

In data processing system 200, ubiquitous high-performance computing(UHPC) server 202 is connected to network 206 along with storage unit208 and client 204. UHPC server 202 may further comprise one or more ofcompute modules 210, storage modules 212, and input/output (I/O) modules214 using interconnect 216. Data processing system 200 may includeadditional servers, clients, storage devices, and network connects notshown. As with network 102 of FIG. 1, network 206 may represent aworldwide collection of networks and gateways that use any type ofcommunication protocols to communicate with one another. Additionally,data processing system 200 may also be implemented to include a numberof different types of networks, such as for example, an intranet, alocal area network (LAN), a wide area network (WAN), or the like. FIG. 2is intended as an example of a UHPC system, not as an architecturallimitation for different embodiments of the present invention, andtherefore, the particular elements shown in FIG. 2 should not beconsidered limiting with regard to the environments in which theillustrative embodiments of the present invention may be implemented.

FIG. 3 depicts one exemplary configuration of a symmetric multiprocessor(SMP) system in the form, of processor node 300 in accordance with anillustrative embodiment. Processor node 300 may contain one or more ofservice processor 302, I/O hubs 306, fabric expansion port 308, andoff-node fabric expansion ports 310. Fabric expansion port 308 andoff-node fabric expansion ports 310 provide connectivity for A and Bports 312 from each of multi-chip modules (MCM) 314 to multi-chipmodules on other processor nodes. Fabric ports X, Y, and Z 316interconnect multi-chip modules 314 within processor node 300.

Additionally, stacked memory chips 323 provide processor memory at eachMCM 314. Each of multi-chip modules 314 may be identical in its hardwareconfiguration but configured by firmware during system initialization tosupport varying system topologies and functions as, e.g. enablement ofmaster and slave functions or connectivity between various combinationsof multiple nodes in a scalable multi-node symmetric multi-processorsystem.

Within a particular multi-chip module 314 there may be found processorunit 320 that may comprise one or more processor cores. Processor node300 may have one or more oscillators 324 routed to each chip found onprocessor node 300. Connections between oscillators 324 and functionalunits extend throughout the board and chips but are not shown in FIG. 3.Similarly, it is understood that many convoluted interconnects existbetween fabric expansion port 308, off-node fabric expansion ports 310,and I/O hubs 306 to the various chips on the board, such as A and Bports 312 and I/O ports 326 of multi-chip module 334, among othercomponents, though such interconnects are not shown in FIG. 3. Theconfiguration illustrated in FIG. 3 is just one example of a processingnode configuration as is merely shown for illustration purposes. One ofordinary skill in the art would recognize that other processing nodeimplementations may be used in the illustrative embodiments withoutdeparting from the spirit and scope of the invention.

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 1-3 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 1-3. Also, theprocesses of the illustrative embodiments may be applied to amultiprocessor data processing system, other than the SMP systemmentioned previously, without departing from the spirit and scope of thepresent invention.

Again, the illustrative embodiments provide a ubiquitoushigh-performance computing (UHPC) system that packages the thousands ofcomponents of a high-performance computing (HPC) system intobuilding-block modules that may be coupled together to form aspace-optimized and energy-efficient product. In a first embodiment, amodular processing device is composed of a plurality of identicalprinted circuit boards and processing nodes housed in identicalprocessor packages referred to as processing modules. Each processingmodule comprises memory, processing layers, and connectivity to power,other processing nodes, storage, input/output (I/O), or the like. Theconnectivity may be provided through wire, wireless communication links,or fiber optic cables and/or interconnects. In the processing module,various heatsink designs remove heat from the components on eachprocessing node. In addition to the processing module, storage and I/Omodules are also provided in similarly formed modules. The storage andI/O modules may be composed of a plurality of printed circuit boardsmounting solid state storage devices and/or optical interconnects. Thephysical design of the modules offers advantages in communicationbandwidth, cooling, and manufacturing costs.

In other embodiments, the heatsink designs are a composite design thatis comprised of two components: the per processing node coolingcomponent and the common core component. Each heatsink component ismounted directly on one or more processing nodes. Since air flow tendsto follow the path of least resistance, one heatsink design fills amajority of the air space such that the flow of air passes between thefins of the heatsink increasing the heat exchange surface area. Inanother heatsink design, the sizing of the heatsink allows the removalof the heatsink and the processing node from the processing module,while the three other heatsinks remain in place. To enable this type ofheatsink design, an empty space is left in the center of the module.Since air flow tends to follow the path of least resistance, toeliminate the loss of beneficial air flow over the heatsinks, a core isinserted into the empty area of the module to fill the air gap,increasing the heat exchange surface area of the heatsinks. The core maybe either a solid core that air flows around, increasing the airpressure on the board mounted heatsinks, or may be another heatsink thatincreases the heat exchange surface area. Since the core is removable,it is still possible to perform in place maintenance tasks on themodule, without dissembling the module.

In another embodiment, the modules are combined to create a scalablespace optimized and energy efficient ubiquitous high performancecomputing (UHPC) system. The UHPC system reduces communication cost,reduces cooling cost, provides reliable operation, and facilitatesmaintainability. The UHPC system does so by using a modular design,where processing nodes are built as modules and assembled as ahexahedron (non-regular cube) according to the computing needs of theend-user. This arrangement results in a reduced distance for thecommunication links, which allows an all-to-all solution.

In still another embodiment, the processing, storage, and/or I/O modulesare constructed such that the modules are liquid tight and are thenliquid cooled in order to increase heat dissipation. Using liquidcooling provides for more modules to be placed in a UHPC system. Inorder to cool the liquid flowing through the modules, heat exchangersare coupled to the external surfaces of a UHPC system. Pumping themodule coolant between the modules and the heat exchangers circulatesthe module coolant through the heat exchange elements. Using theexternal surface of the UHPC system allows heat to be dissipated usingambient air.

While the following embodiments are described with relation to a moduleof cubical design, the illustrative embodiments are not limited to onlya cubical design. That is, other three-dimensional geometricconfigurations may also be used, such as a rectangular box, withoutdeparting from the spirit and scope of the present invention.

FIG. 4A depicts an example of one side of a processing module inaccordance with an illustrative embodiment. In FIG. 4A, processingmodule side comprises one or more of processing nodes 404 coupled tocircuit board 406. Each of processing nodes 404 may comprise memory,processing layers, and connectivity to other ones of processing nodes404 either coupled directly to circuit board 406 or coupled viaconnectors 408 to processing nodes on other circuit boards. Processingnodes 404 may be coupled directly to circuit board 406 in a manner inwhich if one of processing nodes 404 fail, the processing node mayremoved and replaced with a functional processing node. Similarly,circuit board 406 may also be coupled to processing module side 402 in amanner in which if circuit board 406 fails, the entire circuit board 406may removed and replaced with a functional circuit board 406. Each ofconnectors 408 may be any type of connector that provides connectivityto power, other circuit boards, storage, input/output (I/O), or thelike. The connectivity provided by connectors 408 may be wire, fiberoptic, or the like.

FIG. 4B depicts an exemplary cubical processing module 400 in accordancewith an illustrative embodiment. In FIG. 4B processing module 400partially constructed showing three processing module sides 402 a, 402b, and 402 c coupled together. As shown in FIG. 4B, processing moduleside 402 b is coupled to processing module side 402 c via connectors,such as connectors 408. Furthermore, processing module side 402 a isshown to have exterior connector 414 for interfacing with a backplane ofa ubiquitous high performance computing (UHPC) system, which will bedescribed in detail below. While only one of exterior connector 414 isshown, depending on implementation, more than one of external connector414 may be required for interfacing to the UHPC system.

FIG. 4C depicts an exemplary storage module in accordance with anillustrative embodiment. Storage module 420 may comprise storagecontroller 422 mounted to storage module side 424. Storage module 420may also comprise one or more of storage cards 426, which each comprisea plurality of storage devices 428 for storing data, such as storageclass memory chips, along with card specific controller chips 436 forinterfacing with storage controller 422. Each of storage cards 426 maybe coupled to storage card interface 430 which may be coupled to storagecontroller 422 via connector 432 on storage card interface 430 andconnector 434 on storage controller 422. Furthermore, storage moduleside 424 may have one or more exterior connectors 438 for interfacingwith a backplane of a ubiquitous high performance computing (UHPC)system, which will be described in detail below.

FIG. 4D depicts an exemplary input/output (I/O) module in accordancewith an illustrative embodiment. I/O module 440, which may also bereferred to as a network module, provides connectivity for the UHPCsystem to the outside world. I/O module 440 may comprise a plurality ofnetwork interface cards 442 as well as one or more adapters 444 mountedto I/O module side 446. Each of network interface cards 442 may furthercomprise a plurality of pass-thru optical connections 448, which may beused to connect multiple levels of modules in the UHPC system together.Each of network interface cards 442 may also comprise a plurality ofvery high speed Ethernet or Infiniband connectors 450.

FIG. 4E depicts an exemplary filler module in accordance with anillustrative embodiment. Filler module 460 comprises top and bottomrotating slides 462 in top 464 and bottom 466, respectively, to controlthe airflow thru filler module 460 when surrounded by solid sides 468.Rotating slides 462 provide for different airflow impedances so thatdepending on the position the filler module is used within the UHPCsystem, rotating slides 462 may be adjusted to mimic the airflow ofprocessing module 400, storage module 420, or I/O module 440. That is,rotating slides 462 of filler module 460 may be adjusted to providedifferent airflow impedances such that the airflow impedance ofdifferent module types may be matched. This ensures that when fillermodule 460 is used, air flow properties do not change in other areas ofthe UHPC system.

FIG. 5A depicts an exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment. Asopposed to heatsinks that have common sized fins, heatsink 502 of aprocessing module is shaped such that fins 504 on the far edges ofheatsink 502 are shorter in length than a length of fin 506 in themiddle of heatsink 502. Heatsink 502 is constructed in a manner suchthat within processing module 500 with four heatsinks, the heatsinksfill a majority of the air space within the middle of processing modulesuch that the flow of air passes between the fins of the heatsinkincreasing the heat exchange surface area. As is also shown in FIG. 5A,heatsink 502 is of a width and depth such that heatsink 502 covers allprocessing nodes 508 on processing module side 510. Heatsink 502 may beheld in place over processing nodes 508 using generally knownrestrictive methods, such as retaining clips, screws, or the like.Dimension orientation 512 depicts height, width, and depth with relationto the description of heatsink 502. Heatsink 502 may be constructed fromeither copper, aluminum, or another thermally conductive material.

However, in an alternative embodiment (not shown), in order to providefaster access to processor nodes 508 during maintenance, heatsink 502may be of a width and depth to cover only one portion of processor nodes508, which would require another one of heatsink 502 to cover the otherportion of processor nodes 508 such that the majority of the air spacewithin the middle of processing module is still filled such that theflow of air passes between the fins of the heatsink increasing the heatexchange surface area. For example, one smaller depth heatsink may covertwo processor nodes while a similar smaller depth heatsink covers twoother processor nodes. While the illustrative embodiments show four ofprocessor nodes 508 on processing module side 510, the illustrativeembodiments recognize that more or fewer processing nodes may beimplemented such that the width and depth of heatsink 502 requireschanging while the height of heatsink 502 in conjunction with other oneof heatsink 502 still fill a majority of the air space within the middleof processing module such that the flow of air passes between the finsof the heatsink increasing the heat exchange surface area. FIG. 5Bdepicts an example of a constructed processing module that illustratesthe majority of the air space within the middle of processing module 500being filled in accordance with an illustrative embodiment.

FIG. 6A depicts another exemplary heatsink design that may be used in aprocessing module in accordance with an illustrative embodiment. Again,as opposed to heatsinks that have a common sized fins, heatsink 602 ofprocessing module 600 is shaped such that a plurality of fins 604 towardthe ends of heatsink 602 are shorter in length than a plurality of fins606 in the middle of heatsink 602. Heatsink 602 is of a width and depthsuch that heatsink 602 covers all processing nodes 608 on processingmodule side 610. Heatsink 602 may be held in place over processing nodes608 using generally known restrictive methods, such as retaining clips,screws, or the like. Dimension orientation 612 depicts height, width,and depth with relation to the description of heatsink 602.

However, in an alternative embodiment (not shown), in order to providefaster access to processor nodes 608 during maintenance, heatsink 602may be of a width and depth to cover only one portion of processor nodes608, which would require another one of heatsink 602 to cover the otherportion of processor nodes 608. For example, one smaller depth heatsinkmay cover two processor nodes while a similar smaller depth heatsinkcovers two other processor nodes. While the illustrative embodimentsshow four of processor nodes 608 on processing module side 610, theillustrative embodiments recognize that more or fewer processing nodesmay be implemented such that the width and depth of heatsink 602requires changing. Heatsink 602 is constructed in a manner such thatwithin processing module 600 with four of heatsink 602, the sizing ofthe heatsinks allows the removal of the heatsink 602 and circuit board603 from the processing module side 610, while the three other heatsinksremain in place, as is shown in FIG. 6B. Circuit board 603 may becoupled to processing module side 610 by a mechanical mechanism, amagnetic mechanism, or the like, such that circuit board 603 may beeasily removed and replaced. However, to enable this design of heatsink602, an empty space is left in the center of processing module 600.

FIG. 6C depicts a core that may be inserted into an empty space in thecenter of a processing module in accordance with an illustrativeembodiment. Core 620 may be inserted into the empty area of processingmodule 600 to fill the air gap left by the use of four of heatsinks 602.Core 620 may be either a solid or impervious core that air flows aroundand across heatsinks 602 or may be another heatsink (pervious) thatincreases the heat exchange of heatsinks 602. If core 620 is animpervious core, then core 620 may be comprised of non-heat conductivematerial, such as rubber, plastic, or the like. If core 620 is of apervious to airflow design in order to provide added heat exchange toheatsinks 602, then core 620 may be comprised of a thermally conductivematerial, such as copper, aluminum, or the like.

If core 620 is to provide additional heat exchange, then core 620 may becomprised of multiple core sections 622 as shown in overhead view 624.Core sections 622 may be attached in various methods so that onceinserted in the empty area between heatsinks 602 in processing module680, core 620 may expand to maintain thermal conduction of each ofheatsinks 602. In one embodiment, core sections 622 may be coupled usingexpansion mechanisms, such that when a plurality of retention fasteners,such as latches, snap rings, pins, or the like, are released at the topand bottom of core 620, expansion mechanisms 630 expand forcing coressections 622 apart and onto heatsinks 602 as in show in expansion view626. When core 620 is to be removed, a user may use the plurality oflatches, snap rings, pins, or the like, to pull core sections 622 backtogether and away from heatsinks 602 so that core 620 may easily beremoved. In another embodiment, core sections 622 may be coupled usingretention mechanisms 632, such that when expansion rod 634 is insertedinto the center of core 620, retention mechanisms 632 are forced toexpand which forces cores sections 622 apart and onto heatsinks 602 asin show in expansion view 628. When the expansion rod is removed fromthe center of core 620 retention mechanisms 632 pull the cores sections622 back together and away from heatsinks 602 so that core 620 mayeasily be removed.

The use of core 620 allows maintenance to be performed on processingmodule 600 without dissembling processing module 600. In order toincrease heat conductivity between heatsinks 602 and core sections 622,the edges of core 620 that will come in contact with heatsink 602 may becoated with a thermal conductive paste prior to being inserted into theempty area between heatsinks 602 in processing module 600.

FIG. 6D illustrates another embodiment of an expanding core inaccordance with an illustrative embodiment. In FIG. 6D, core 640comprises four core sections 642 with a center pin 644. Center pin 644,which may be removable, at the center of core 640 acts as a guide toprevent core 640 from touching the board mounted heatsinks. While notshown, center pin 644 may be affixed to a aerated base plate at thebottom of module 600 or a stack of modules such that when core 640 isinserted into the module or stack of modules, center pin 644 acts as aguide so that the exterior edges of core 640 does not contact anyheatsink in the module. This is especially important to maintain thethermal material on the surfaces of core 640 that will come in contactwith the board mounted heatsinks, preventing the thermal material frombeing wiped out during sliding core 640 into the module. The thermalmaterial and the applied pressure between core 640 and the heatsinks ofthe processing module are both important to maintain high thermalconductivity between the two components for efficient cooling of core640. When wedge 646 is inserted into core 640, core 640 expands so thatthe sides of core 640 come into contact with the heatsinks of theprocessing module as is shown in expansion views 648 and 650.

FIG. 7A depicts an example of a scalable space-optimized andenergy-efficient computing system in accordance with an illustrativeembodiment. In FIG. 7A, a ubiquitous high performance computing (UHPC)system 700 provides a compact arrangement of modules 702 configured inframe 704 that reduces communication cost, reduces cooling cost,provides reliable operation, and facilitates maintainability. Themodular design of UHPC system 700 provides these benefits by assemblingthe modules in a hexahedron (non-regular cube) according to thecomputing needs of the end-user which reduces distance for thecommunication links. UHPC system 700 comprises frame 704, modules 702,air inlet 706, air mixing plenum 708, and one or more cooling fans 710.Each of modules 702 may be either a processing module, a storage module,an input/output (I/O) module, or a filler module, as previouslydescribed, and may be installed in frame 704 similar to a drawer as isillustrated. The other components of UHPC system 700 will now bedescribed in detail.

FIG. 7B depicts an exemplary frame of a UHPC system in accordance withan illustrative embodiment. Frame 704 provides a plurality of identicalmodule compartments 712 such that any type of module may be insertedinto a single one of module compartment 712 and be connected viabackplane 714 to power, storage, communication, or whatever access isrequired by the module. Frame 704 provides sections between each ofmodule compartments 712 so that cabling may be run between the variousconnectors of the backplanes as well as to external power and networkconnects for environments where UHPC system 700 is deployed.Additionally, the top and bottom of each of module compartments 712 areopen so that air may flow through each column of module compartments 712from the air inlet 706 to air mixing plenum 708. Each level of modulecompartments may also be individual sections such that UHPC system 700may comprise as few as one level up to any number of levels such thatthe power and cooling needs are still met by UHPC system 700.

Air inlet 706 may be a compartment that has a solid bottom 718 with opensides and top. Each of the sides of air inlet 706 may be constructedsuch that access panels 720 provide for the insertion and replacement ofair filters 722. Air would flow through the air filters in the sides ofair inlet 706 and up through the top of air inlet 706 through modulecompartments 712 to air mixing plenum 708. The top of air inlet 706 maybe constructed in a way that the top section provides knock outs 724 inthe dimensions of module compartments 712 so that a user may remove onlythose knock outs 724 for those columns of module compartments 712 thatare populated in frame 704. Using knock outs 724 in air inlet 706 allowsthe user to cool only those areas of frame 704 that are occupied bymodules 702. Further, in the event a knock out 724 is erroneouslyremoved or if modules 702 are removed such that a column of modulecompartments 712 no longer has any occupying modules 702, filler platesmay be provided to replace the knock out 724.

FIG. 7C depicts an exemplary top down view of a UHPC system inaccordance with an illustrative embodiment. As can be seen in FIG. 7C,air mixing plenum 708 may be a sectional compartment that is placedabove the top level of module compartments 712 and covers the outsideperimeter of frame 704 such that air flowing up through modulecompartments 712 will be accumulated in the area of air mixing plenum708. Also show in FIG. 7C is center area 716 which is shown as empty butmay be used for cabling, other modules, or the like. The use of centerarea 716 for modules is illustrated in a different embodiment that isdescribed below. In this illustration, center area 716 may not have airflow as restricted by air inlet 706 previously described.

FIG. 7D depicts an exemplary view of the cooling fans of a UHPC systemin accordance with an illustrative embodiment. In FIG. 7D there areshown four cooling fans 710 each of which draw air though the air inletat the bottom of the UHPC system, through one or more modulecompartments, and through the air mixing plenum. Each fan 726 of fans710 may be controlled either individually or as a group. That is,dependent on sensed temperature in the UHPC system, each fan 726 of fans710 may be controlled such that one fan turns on when the temperatureexceeds one threshold temperature and the other fans may individuallyturn on as other temperature associated thresholds are exceeded.Likewise, each fan 726 of fans 710 may individually turn off astemperature levels within the UHPC system decrease and the associatedtemperature thresholds are no longer exceeded. The temperaturethresholds may be controlled through simple thermostats 728 associatedwith each fan 726 or other more complex thermal controls. In analternative embodiment, fans 710 may be a single fan 726 that has avariable motor 730 that increases to draw more air as the temperature ofthe UHPC system increases.

Additionally, while the exemplary embodiment illustrates four of fans710, the illustrative embodiment recognizes that more or fewer fans maybe used without departing from the spirit and scope of the invention.Further, while fans 710 are shown on top of the UHPC system, fans 710may be placed anywhere in the UHPC system such that air is pushed orpulled through the UHPC system. For example, fans 710 may be locatedbelow the air inlet, or between the air inlet and the modulecompartments.

FIG. 8A depicts an exemplary module that is liquid tight and liquidcooled in order to increase heat dissipation in accordance with anillustrative embodiment. In FIG. 8A, module 800 is constructed similarto the modules previously described but also has a top and bottom sidethat causes the module to be liquid tight other than inlet port 802 andoutlet port 804, as well as electrical and optical connections. Inletport 802 and outlet port 804 may be located at opposing locations onmodule 800, such that module 800 may be filled with a non-conductiveliquid and pump 812 located within or near the module may pump theliquid through module 800. As devices within the module heat thenon-conductive liquid, the hot liquid flows out of or exits module 800via outlet port 804 into exit tube 808 and through heat exchanger 806which may be located on and coupled to an outside air exposed area ofthe UHPC system, preferably the module with which the heat exchanger isassociated. Exposure of the liquid to the ambient air around the UHPCsystem through heat exchanger 806 cools the liquids such that, after theliquid finishes its pass through heat exchanger 806, the liquid isreturned through return tube 810 back into module 800 via inlet port 802at a cooler temperature than when it exited module 800. While pump 812is shown to be located in line with exit tube 808, pump 812 may belocated either in the exit line or the return line, whichever isdetermined to be more efficient. The pump may also be located in acavity within the heat exchanger, dedicated to host the pump, in orderto make it easily accessible for repair and routine pump maintenance.

While not shown in FIG. 8A, the components within module 8A may becooled with the use of a cold plate mounted on each circuit board, suchas circuit board 603 in FIG. 6B, replacing a heatsink, such as heatsink602 of FIG. 6B. Coolant may be circulated through each cold plate from acommon inlet port 802 and out a common outlet port 804. In thisembodiment the cooling fluid may be conducting and filling the modulewith cooling fluid is not required. Thus, module 800 may not need to beliquid tight. In this embodiment only the interior cooling components ofthe module are different than the other embodiments discussed.

FIG. 8B depicts an exemplary cooling of multiple modules by a singleheat exchanger in accordance with an illustrative embodiment. In thisembodiment, heat exchanger 806 provides sufficient cooling such that theliquid may be passed through more than one of modules 800 before beingcooled again in heat exchanger 806. In this embodiment, two or more ofmodules 800 are configured such that outlet port 804 of a first module800 is coupled to the inlet of its heat exchanger 806 via exit tube 808and the outlet of heat exchanger 806 is coupled to inlet port 802 of asecond module 800 via return tube 810. Then outlet port 804 of thesecond module 800 is coupled to inlet port 802 of the first module 800using coupling tube 814.

FIG. 8C depicts an exemplary structure where the center area of a UHPCsystem is populated by one or more modules. In this illustrativeembodiment, liquid tight center modules 820 may be inserted within agiven module compartment and coupled to a backplane in order to obtainconnectivity to power, communications, storage, or the like. Since thesecenter modules 820 do not have direct access to an outside air exposedarea of the UHPC system, extension tubes 815 are used to give the centermodules 820 access to the heat exchangers 806 on the exterior perimeterof the UHPC system. In this illustrative embodiment, concern is givenwith regard to the length of the exit tube and the return tube (showncombined as extension tubes 815) associated with each of modules 800 andcenter modules 820 so that the shortest distance to and from the heatexchanger is provided.

FIG. 8D depicts how a three-dimensional very-large-scale integration(VLSI) global routing technique may be applied to generate a layout forthe tubes, such as exit tubes 808, return tubes 810, coupling tube 814,and extension tube 815 of FIGS. 8A-8C, in accordance with anillustrative embodiment. To solve the problem of routing the heatexchangers to the modules, simple graph 820 is constructed thatrepresents the topology of the modules. There is one vertex 822 permodule and a single undirected edge 821 that connects the modules, whichabut each other. Edges 821 represent the channels through which tubesare allowed to pass. Each edge 821 is given a capacity which representsthe number of tubes which are allowed to pass between adjacent modules.In conventional VLSI global routing, connections of the components areknown as a priori, hence fixed. For the routing of the tubes in a UHPCsystem, there is only one known endpoint per connection and a heatexchanger must connect to each module which requires cooling. To adaptthe VLSI global routing techniques to this domain, a unique air vertex823 is added, which represents the air where heat is eventuallyradiated. From air vertex 823, undirected edges 821 are added to all ofthe possible locations where the heat exchangers may be attached. Foreach of edges 821 that connect air vertex 823 to the 3D mesh, a capacityis assigned that is equal to the number of modules with which the heatexchanger is allowed to be connected. For the remainder of edges 821,their capacities are set as the number of tubes, such as exit tubes 808,return tubes 810, coupling tubes 814, and extension tube 815 of FIG.8A-8C, that are allowed to pass between the adjacent modules inquestion. Each module 800 which needs to be cooled may be connected withthe air vertex 823. A solution to this routing problem routes tubes tothe heat exchangers 806.

FIG. 8E depicts another exemplary structure where the heat exchangersalong the walls of a module are replaced by heat exchanger panels inaccordance with an illustrative embodiment. In FIG. 8E, heat exchangerpanels 832 and 833, which may be pervious or impervious, on the outsideof UHPC system 830 represent panels of heat exchangers that may be usedto cool the non-conductive liquid coolant that circulates through aplurality of modules 800. Exit tube 808 and return tube 810 transfer thenon-conductive liquid coolant to the heat exchanger panels 832 and 833.Extension tubes 815 transfer the non-conductive liquid coolant throughmodules 800. A particular UHPC system 830 may have heat exchanger panel832 and/or 833 coupled to a single module 800, in which case there is noneed to use extension tubes 815. Each of heat exchanger panels 832 and833 may be coupled to the surface of UHPC system 830 such that heatexchanger panels 832 and 833 are adjacent to the sides of UHPC system830, as is shown by heat exchanger panel 832, or each of heat exchangerpanels 832 and 833 may be coupled to the surface of UHPC system 830 suchthat heat exchanger panels 832 and 833 may be tilted away from the sidesof UHPC system 830 so that air flow may be increased across the heatexchangers, as is shown by heat exchanger panel 833. Furthermore,cooling of the non-conductive liquid coolant may be facilitated byassisting the flow of air around heat exchanger panels 832 and 833. Forthat purpose, heat exchanger panels 832 and 833 may be pervious orimpervious to air flow.

FIG. 8F depicts a heat exchanger panel in accordance with anillustrative embodiment. In FIG. 8F, heat exchanger 806 is installed inthe heat exchanger panel 832. Exit tube 808 and return tube 810 are usedto allow the non-conductive liquid coolant to cool. The area of the heatexchanger panel 832 may be smaller, equal to, or larger than the area ofthe exterior of the UHPC system 830. Heat exchanger panel 832 mightdissipate heat on one or many of its sides. For example, heat exchanger832 that is adjacent to UHPC system 830 may be designed to onlydissipate heat by the outer surface. While heat exchanger 833 that istilted with respect to UHPC system 830 might dissipate heat by the innerand the outer surfaces.

Thus, the illustrative embodiments provide a ubiquitous high-performancecomputing (UHPC) system that packages the thousands of components of ahigh-performance computing (UHPC) system into building-block modulesthat may be coupled together to form a space-optimized andenergy-efficient product. The illustrative embodiments also provide forvarious heatsink designs that enable an elegant assembly and in placemaintenance for the heatsink and the module, while maintaining largeeffective heat exchange area and high pressure for efficient cooling.The illustrative embodiments also provide for an alternative to aircooling using a liquid cooling system with coolant/air heat exchangingenabled by skin heat exchangers mounted either on the interior or theexterior surface of the UHPC system.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A scalable space-optimized and energy-efficient computing systemcomprising: a plurality of modular compartments in at least one level ofa frame, the frame configured in a hexahedron configuration, the framehaving at least one modular compartment of the plurality of modularcompartments on each of four sides of the frame; an air inlet; an airmixing plenum; and at least one fan, wherein: the plurality of modularcompartments are affixed above the air inlet, the air mixing plenum isaffixed above the plurality of modular compartments, the at least onefan is affixed above the air mixing plenum, and when at least one moduleis inserted into the one of the plurality of modular compartments, theat least one module couples to a backplane within the frame, wherein thebackplane has four portions and wherein each portion of the fourportions of the backplane is parallel to a face of one side of the foursides of the frame having the at least one modular compartment.
 2. Thecomputing system of claim 1, wherein the at least one module is at leastone of a modular processing module, a storage module, an input/output(I/O) module, or a filler module.
 3. The computing system of claim 2,wherein the at least one module is the modular processing modulecomprising: a set of processing module sides, wherein each processingmodule side comprises: a circuit board; a plurality of connectorscoupled to the circuit board; and a plurality of processing nodescoupled to the circuit board, wherein the each processing module side inthe set of processing module sides couples to another processing moduleside using at least one connector in the plurality of connectors suchthat when all of the set of processing module sides are coupled togetherthe modular processing module is formed, and wherein at least oneprocessing module side of the set of processing module sides has anexterior connector coupled to the circuit board of the at least oneprocessing module side in order to connect to the backplane of thecomputing system.
 4. The computing system of claim 2, wherein the atleast one module is the storage module comprising: a storage controllercoupled to a storage interface card, wherein the storage interface cardis coupled to a plurality of storage circuit boards, wherein eachcircuit storage board of the plurality of storage circuit boardscomprises a plurality of storage devices for storing data and aplurality of controller chips, and wherein the storage interface card isfurther coupled to an exterior connector in order to connect to thebackplane of the computing system.
 5. The computing system of claim 2,wherein the at least one module is the I/O module comprising: aplurality of adapters coupled to a plurality of network interface cards,wherein each network interface card of the plurality of networkinterface cards comprises a plurality of pass-thru optical connectionsfor connecting multiple levels of modules in the computing system andwherein each network interface card comprises a plurality of very highspeed Ethernet or Infiniband connectors.
 6. The computing system ofclaim 2, wherein the at least one module is the filler modulecomprising: four solid sides; a top and a bottom, wherein both the topand the bottom each have rotating slides, wherein the rotating slidesare adjustable to provide different airflow impedances such that theairflow impedance of different module types are matched, wherein thedifferent module types are at least one of the modular processingmodule, the storage module, or the input/output (I/O) module.
 7. Thecomputing system of claim 1, wherein the plurality of modularcompartments are formed such that when any column of modules areinstalled in the computing system, air will flow through each of themodules providing cooling to each module.
 8. The computing system ofclaim 1, wherein the air inlet is a compartment that has a solid bottomwith open sides and an openable top, wherein the open sides of the airinlet are constructed such that access panels provide for insertion andreplacement of air filters, wherein a top of the air inlet isconstructed in a way that a top section provides knock outs indimensions of the plurality of module compartments, and wherein a userremoves only those of the knock outs for selected columns of theplurality of module compartments that are populated in the frame.
 9. Thecomputing system of claim 1, wherein the at least one fan is a pluralityof fans each individually controlled by thermostats with differentthermal thresholds.
 10. The computing system of claim 1, wherein the atleast one fan in a plurality of fans has a variable motor that increasesin speed to draw more air as a temperature of the computing systemincreases.
 11. A scalable space-optimized and energy-efficient computingsystem comprising: a plurality of modular compartments in at least onelevel of a frame, the frame configured in a hexahedron configuration,the frame having at least one modular compartment of the plurality ofmodular compartments on each of four sides of the frame, wherein atleast one module is inserted into one of the plurality of modularcompartments, the at least one module is coupled to a backplane withinthe frame, wherein the backplane has four portions, wherein each portionof the four portions of the backplane is parallel to a face of one sideof the four sides of the frame having the at least one modularcompartment, and wherein the at least one module is at least one of amodular processing module, a storage module, an input/output (I/O)module, or a filler module.
 12. The computing system of claim 11,further comprising: an air inlet; an air mixing plenum; and at least onefan, wherein: the plurality of modular compartments are affixed abovethe air inlet, the air mixing plenum is affixed above the plurality ofmodular compartments, and the at least one fan is affixed above the airmixing plenum.
 13. The computing system of claim 12, wherein the airinlet is a compartment that has a solid bottom with open sides and anopenable top, wherein the open sides of the air inlet are constructedsuch that access panels provide for insertion and replacement of airfilters, wherein a top of the air inlet is constructed in a way that atop section provides knock outs in dimensions of the plurality of modulecompartments, and wherein a user removes only those of the knock outsfor selected columns of the plurality of module compartments that arepopulated in the frame.
 14. The computing system of claim 12, whereinthe at least one fan is a plurality of fans each individually controlledby thermostats with different thermal thresholds.
 15. The computingsystem of claim 12, wherein the at least one fan is in a plurality offans has a variable motor that increases in speed to draw more air as atemperature of the computing system increases.
 16. The computing systemof claim 11, wherein the at least one module is the modular processingmodule comprising: a set of processing module sides, wherein eachprocessing module side comprises: a circuit board; a plurality ofconnectors coupled to the circuit board; and a plurality of processingnodes coupled to the circuit board, wherein the each processing moduleside in the set of processing module sides couples to another processingmodule side using at least one connector in the plurality of connectorssuch that when all of the set of processing module sides are coupledtogether the modular processing module is formed, and wherein at leastone processing module side of the set of processing module sides has anexterior connector coupled to the circuit board of the at least oneprocessing module side in order to connect to the backplane of thecomputing system.
 17. The computing system of claim 11, wherein the atleast one module is the storage module comprising: a storage controllercoupled to a storage interface card, wherein the storage interface cardis coupled to a plurality of storage circuit boards, wherein eachcircuit storage board of the plurality of storage circuit boardscomprises a plurality of storage devices for storing data and aplurality of controller chips, and wherein the storage interface card isfurther coupled to an exterior connector in order to connect to thebackplane of the computing system.
 18. The computing system of claim 11,wherein the at least one module is the I/O module comprising: aplurality of adapters coupled to a plurality of network interface cards,wherein each network interface card of the plurality of networkinterface cards comprises a plurality of pass-thru optical connectionsfor connecting multiple levels of modules in the computing system andwherein each network interface card comprises a plurality of very highspeed Ethernet or Infiniband connectors.
 19. The computing system ofclaim 11, wherein the at least one module is the filler modulecomprising: four solid sides; a top and a bottom, wherein both the topand the bottom each have rotating slides, wherein the rotating slidesare adjustable to provide different airflow impedances such that theairflow impedance of different module types are matched, wherein thedifferent module types are at least one of the modular processingmodule, the storage module, or the input/output (I/O) module.
 20. Thecomputing system of claim 11, wherein the plurality of modularcompartments are formed such that when any column of modules areinstalled in the computing system, air will flow through each of themodules providing cooling to each module.